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ABSTRACT 



New ways are proposed to look at data from the Canadian 
School Achievement Indicators Program (SAIP) using item response theory (IRT) 
modeling. The focus is on the traditional test of the SAIP 1997 mathematics 
study. The test is two-staged in that the first 15 items, of median 
difficulty, were to be completed beforehand by all students as a "placement" 
test. Where students were to start the second portion of the test was 
determined by their scores on the first 15 items. The dimensionality of the 
scale, the item fit, item bias, and item invariance were considered in 
applying IRT modeling. The scale obtained reflected the relative ability of 
the students. Results suggest that mean scale scores can be safely compared 
between jurisdictions, genders, or age groups. The proposed approach seems 
reasonably appropriate when a two-stage testing procedure has been used to 
evaluate students' performance. (Contains 1 table, 5 figures, and 19 
references.) (SLD) 
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Introduction 

_ . tha , ,he Canadian Council of Ministers of Education approved the implementation of a national 

It was about ten yean > age > Achieveme nt Indicators Program. The aim of the program was to undertake 

assessment program called • ^- t - . f or 13 and 16-year-old French or English students. 

year 2000 : the firs, cycle included 

Two cycles of studies were f ® sciei J ce in 1996 ; f or the second cycle, mathematics in 1997, 

mathematics in ! 993 ’ To^fndscience hi 1999. The 1999 science assessment is well under way and the Council has 
aruiounced thTthe nixt cycle will be math assessment for 2001, reading and writing for 2002 and science for 2003. 

it * ii th„ c ATP results had always been reported using classical statistics : proportions of students reaching a 
P f ^orln^evd fS each age ^roup (13 and 16 year-old), each gender and each jurisdiction (in Canada there 
££££££ Tories (lift .he new Nunavut)). The main objective of due paper is to propose new ways to 

look at the SAIP data using item response modelling. 



SAIP features for the 1997 math assessment 

. t _ r * f nf u;^u ic tn pvaluate the level of performance for Canadian students in the three 
h etween cycles of studies, changes in perfomunces for each subject 

matter, that is to evaluate the so-called trend indicators. 

The usual SAfP samphng design involves, 

TVS of sampling being the sch^ls, 

X^onTote number of Uresmdenut in the schools. The students werechosen a. the second stage of samphng. 

Each SAIP study involves two typKof exmnd^pome^ho^ 

divided in wo subsamples, about hah of Ure whole sample of students 
assigned to each type of achievement variables involved. 

This paper will be dealing uninly with the traditional 

items, M of which were umhiplwhojce^er. wer' so tot level 1 

and 17 statisucs items. Eac *V, lt ^”J^ S j . m harder ones Level 1 items were those from 16 through 40, level 2 items 

5 i,enKftom 101 10 125 



The two-stage testing design 



_ 1<MVf . soldered two-staged in that the first 15 items, of median difficulty, were to be completed 

The 1997 math tes t can > for ^ what is called a placement test (Figure 1). Those having succeeded in 

beforehand by all students. These 3 winning with item 16 (the first encountered level 1 item) up to, if 

not more than 10 of those 15 items were carry 13 j f ^ f 1 15 items had to go to item 41 (the first 

possible, item 125 (to las. Lse Sg go. more ton 13 items right staned a. item 66 

(the^usrCTC^ntemd toeO ^ c ^dt^^|c n | s ^^g^^i^’abfe^ldlnts^and ^oup^ to'rorue^bfc^WWirto 

m^pttoSorto 16-year.ld sZJs wen, in group 3. 



